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Abstract 


A college degree is often viewed as a key step toward better employment and higher earnings. 
Many community college students, however, never graduate and cannot reap the financial 
benefits associated with a college degree. Although existing research suggests that financial aid 
interventions can modestly improve students’ short-term academic outcomes, there is little 
rigorous evidence on the critical question of whether such interventions improve graduation 
rates or employment outcomes. This study helps to fill that gap by using a randomized con- 
trolled trial involving over 2,000 community college students. It focuses on low-income parents, 
most of whom are low-income mothers. The study includes four years of post-random assign- 
ment data to examine the long-term impact of a perfonnance-based scholarship program — 
financial aid that is contingent on academic perfonnance — on degree receipt, employment, and 
earnings. The findings provide evidence that the program decreased the time it took students to 
earn a degree, but the findings do not provide evidence that the program increased employment 
or earnings by the end of the study period. 
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Introduction 

A college degree is often viewed as a key step toward better employment and higher earnings. 
Many students who invest their time and money in community colleges, however, never 
graduate and consequently cannot reap the financial benefits associated with a college degree. 
Among first-time students who entered community college during the 2003-2004 academic 
year, for example, roughly two-thirds did not earn a degree or certificate within six years (U.S. 
Department of Education, 2011). Low-income students may find it particularly difficult to 
complete their education and earn a degree, as going to college can require a substantial 
financial commitment relative to their total earnings. For some students, nearly 60 percent of 
household income is needed to cover the total cost of a two-year community college. 1 Many 
students use financial aid to help pay for college, but even so, financial need can still be a barrier 
to finishing school. Low-income students often work because of financial pressures, but 
students who reduce course enrollment in order to work more are less likely to graduate 
(Adelman, 2006). 

Additional sources of financial aid for low-income students may be one solution to in- 
crease graduation rates, and nonexperimental research (studies that use observational data) 
suggests that additional financial aid positively affects enrollment (Bettinger, 2010; Castleman 
and Long, 2013; Cornwell, Mustard and Sridhar, 2006; Dynarski, 2003; Jackson, 2010; Kane, 
2003), persistence (Bettinger, 2004), credit accumulation (Castleman and Long, 2013; Scott- 
Clayton, 2010), and ultimately degree receipt (Dynarski, 2005; Castleman and Long, 2013; 
Scott-Clayton, 2010). These nonexperimental studies are encouraging but they leave open the 
possibility that other factors, such as preexisting differences in academic skills or motivation, 
may have caused the differences in student outcomes, and that additional financial aid may not 
have led to higher graduation rates. 

In a randomized controlled trial, researchers randomly assign students to either a pro- 
gram group that is given the opportunity to receive additional financial aid, or a control group 
not given the opportunity. Such a trial is the strongest design to establish causality, because 
random assignment ensures that no systematic factors determine whether students are offered 
the opportunity to receive additional financial aid. 

Researchers have also used randomized controlled trials to test some financial aid inter- 
ventions. Compared with nonexperimental studies, they generally find more modest effects on 
average academic outcomes (Angrist, Lang, and Oreopoulos, 2009; Leuven, Oosterbeek, and 
van der Klaauw, 2010). A number of the more rigorous evaluations of financial aid interventions 


'This estimate is for students with families in the lowest quintile of income who are enrolled in two-year 
colleges in Ohio. National Center for Higher Education Management Systems (2013). 
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are part of a large-scale demonstration project to evaluate perfonnance-based scholarships — 
additional financial aid that is contingent on academic perfonnance — called the Performance- 
Based Scholarship Demonstration (PBS Demonstration) (Richburg-Hayes et ah, 2009; Cha and 
Patel, 2010; Miller, Binder, Harris, and Krause, 2011; Patel and Rudd, 2012; Patel, Richburg- 
Hayes, De la Campa, and Rudd, 2013; Patel and Valenzuela, 2013; Sommo et al., 2014). To 
date, however, there is little rigorous evidence that answers the critical questions of whether any 
fonn of financial aid for higher education can improve degree receipt, employment, and 
earnings. 

The study described in this article provides insight into precisely those questions. It uses 
a randomized controlled trial involving over 2,000 community college students and it includes 
four years of post-random assignment data on academic and employment outcomes to examine 
the long-term impact of a perfonnance-based scholarship program on degree receipt, employ- 
ment, and earnings. Additionally, this study targeted low-income parents — predominantly low- 
income mothers — a population that may be especially vulnerable to academic challenges 
related to financial constraints (Richburg-Hayes et al., 2009). 


The Ohio Performance-Based Scholarship Program 

Performance-based scholarships were first tested in a randomized controlled trial in Louisiana 
(Richburg-Hayes et al., 2009), and later at other sites across the country through the PBS 
Demonstration (Cha and Patel, 2010; Miller, Binder, Harris, and Krause, 2011; Patel and Rudd, 
2012; Patel, Richburg-Hayes, De la Campa, and Rudd, 2013; Patel and Valenzuela, 2013; 
Sommo et al., 2014). In Louisiana, low-income parents at two New Orleans community 
colleges enrolled in a program that allowed them to earn up to $1,000 per semester for two 
semesters, provided that they maintained at least half-time enrollment and a “C” average or 
better. The evaluation found that the scholarships had positive effects on several outcomes, 
including students’ credit accumulation and semester-to-semester retention. Evidence of these 
effects persisted into the third and fourth semesters, when most students were no longer eligible 
for the scholarship. Just after the program ended, Hurricane Katrina struck the Gulf Coast 
region, causing severe destruction and temporarily shutting down the two colleges. Many 
students in the study moved away, and the devastation inflicted made it virtually impossible to 
determine whether the program had long-term effects on graduation and employment outcomes 
(Richburg-Hayes et al., 2009). 

In Ohio, however, the state legislature was impressed by the short-term outcomes of the 
Louisiana study and developed a similar statewide program for low-income students, using 
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surplus funds from the Temporary Aid to Needy Families (TANF) program. 2 The TANF 
Educational Awards Program (TEAP) was implemented statewide in the 2006-2007 school 
year, as a one-year program that ended when the surplus funds had been spent. In the 2008- 
2009 academic year, the program was reintroduced at three colleges within the context of a 
randomized controlled trial in the PBS Demonstration to provide rigorous evidence about the 
program’s impact. The Ohio Department of Jobs and Family Services (ODJFS) funded the 
scholarship using flexible TANF dollars, and the Ohio Board of Regents (OBR) administered 
the program through its division of State Grants and Scholarships. 

State administrators believed that low-income parents were likely to stay in Ohio after 
graduating from college, so the use of TANF funds fit naturally with the state’s strategic plan of 
producing more college graduates who would remain in the state (Fingerhut, 2008). The study 
population in Ohio was made up mostly of low-income students who were also mothers, with a 
large majority receiving government benefits such as TANF and food stamps. 3 This population 
was also generally older and half of the students were also employed. Like many low-income 
students, the students in this study had additional responsibilities such as work obligations. As 
parents, however, they were often juggling child care responsibilities as well. 

The Ohio program was implemented at Lorain County Community College, Owens 
Community College, and Sinclair Community College, covering three of Ohio’s four geo- 
graphic comers. Located in northeast Ohio’s small city of Elyria, Lorain enrolled over 1 1,000 
students in fall 2008, when the PBS Ohio program started. 4 Owens was almost twice the size 
of Lorain, with 21,000 students in fall 2008, and is located in northwest Ohio, serving 
students in Toledo and neighboring Michigan. Sinclair, located in south Ohio near Dayton, 
served over 19,000 students in fall 2008. Across these three colleges, 2,285 students partici- 
pated in the study. About 60 percent were randomly assigned to the program group and about 
40 percent were randomly assigned to the control group. 5 

The Ohio scholarship program aligned with the general characteristics of programs in 
the PBS Demonstration. These programs are structured with two main goals: first, to make 
college more affordable for low-income students; and second, to structure the scholarship 

2 The information in this section is adapted from Cha and Patel (2010). 

3 In addition to noted criteria, participants were at least 18 years of age, U.S. citizens, residents of Ohio, 
and had not been incarcerated. The main criteria for the program were that students needed to be parents (not 
necessarily custodial), and have an Expected Financial Contribution (EFC) of zero. An EFC of zero was 
approved by the State of Ohio as a proxy for TANF eligibility in this program. EFC was determined using 
institutional data on the Free Application for Federal Student Aid (FAFSA) at the time of random assignment, 
but may have changed at a later date. 

4 A11 enrollment numbers are from the U.S. Department of Education, National Center for Education Sta- 
tistics, Integrated Postsecondary Education Data System (2008). 

5 For more information on the implementation of the PBS program in Ohio, see Cha and Patel (2010). 


3 



payments to provide incentives for behaviors associated with good academic progress. The key 
components of the scholarship are: 

1. Performance-based scholarships: Awards are paid if students meet basic condi- 
tions regarding enrollment and grades in college courses. They thus act as incen- 
tives, rewarding behavior associated with academic success. 

2. Current semester focus: Scholarships are based on academic performance in the 
current semester, regardless of performance in previous semesters or high school 
performance. This feature is unlike merit-based aid in which students have to first 
qualify based on high school performance (for example, high school grade point 
average (GPA)) or grades from a previous college term. 

3. Paid directly to students: Often scholarships are paid directly to institutions. In 
contrast, students in the performance-based scholarship program receive the money 
directly and can use it to cover any expenses, including those that could derail con- 
tinued attendance and success (for example, child care or transportation). 

4. Designed to supplement other financial aid: Performance-based scholarships are 
meant to supplement Pell Grants and state aid to help meet the needs of low-income 
students. The intervention gives students more money to cover academic and living 
expenses, and can potentially reduce their dependency on loans. 

The performance-based scholarship program in Ohio offered a scholarship for two con- 
secutive semesters at Lorain and Owens, and three consecutive quarters at Sinclair. 6 Students 
were given a full-time award of $900 per semester or $600 per quarter for achieving a “C” or 
better in 12 or more credits; or a part-time award of $450 per semester or $300 per quarter for 
achieving a “C” or better in 6 to 1 1 credits. At each institution, regardless of the semester or 
quarter system, students were eligible for a maximum award of $1,800 over the entire program 
duration. The differential award depending on attendance level was designed to be more 
flexible for this population, and to allow students to attend part time and still earn part of the 
scholarship. 

Students who participated in the program could earn any combination of part-time 
award, full-time award, or no award for the duration of the program. Students were paid only 
once and at the end of each semester or quarter to avoid interference with their eligibility for 
public benefits, since almost 70 percent of the sample was receiving some sort of public 


6 Quarter institutions have three quarters per academic year, while semester institutions have two semes- 
ters. To adjust for inflation due to the additional term at quarter institutions, credits attempted and earned at 
quarter institutions are multiplied by two-thirds. 
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benefits. As a result, students received regular reminders about the scholarship throughout the 
semester or quarter by e-mail and postcard. These reminders were designed to be positive, 
informative, and regular, and to keep the scholarship at the forefront of students’ minds. 


The Theory of Change 

The program in Ohio was intended to determine whether a scholarship-only program (that is, 
a program without any added services such as advising or tutoring) could lead to positive 
outcomes for students . 7 The scholarship opportunity and the reminders about the scholarship 
were designed to influence behaviors that might lead to improvements in student outcomes, 
including: 

• Motivation: The scholarships were designed to provide incentives for aca- 
demic progress and to motivate students. Making payments contingent on 
key academic benchmarks was intended to encourage students to progress 
academically and motivate them to perform at higher levels. 

• Change in effort or reallocation of priorities: In order to meet the scholar- 
ship benchmark, students might study more (or better), work less, or pay for 
child care so that they could focus more on their studies. 

• Enroll in more classes: Having additional funds from a scholarship earned 
in the first semester or quarter might allow students to enroll in more classes 
in following semesters or quarters. 

• Reduce financial stress: Additional funds might also relieve some of the fi- 
nancial stresses facing low-income parents. 

• Increased confidence: Positive messages and occasional interaction with the 
scholarship coordinator might boost students’ confidence or simply remind 
them of the requirements to get the award. 


Research Design and Data 

This study uses a randomized controlled trial to estimate the impact of the intervention. The 
estimates are intent-to-treat estimates — the estimated effect of being offered the opportunity to 
participate in the scholarship program — calculated by comparing the average outcomes of all 


7 The programs in Louisiana and at some other colleges in the PBS Demonstration included an advising, 
tutoring, or another student service component. 
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students randomly assigned to the program group with the average outcomes of all students 
randomly assigned to the control group. The study is designed to estimate the impact of the 
program as a whole, rather than to disentangle the effects of the scholarship from the effects of 
the reminders or scholarship coordinators at the colleges . 8 

Data 

Student data were collected from several sources. Students provided demographic data 
through a Baseline Information Form (BIF), administered prior to random assigmnent. Finan- 
cial infonnation was collected from Free Application for Federal Student Aid (FAFSA) data 
provided by the Ohio Board of Regents (OBR) and financial aid records provided by each of the 
partner institutions. FAFSA and BIF data are used to assess recruitment, describe the sample, 
and compare students in the program and control groups on observable characteristics. The 
financial aid records are used to assess whether students in the program group received more 
financial aid through the scholarship program, compared with students in the control group. 

Students in both the program and control groups were also surveyed during either the 
second or third semester or quarter of the program for each cohort . 9 The survey covered a range 
of factors hypothesized to mediate the impact of the program on student outcomes, including 
attendance, study habits, employment, and motivation. Students assigned to the program group 
were also asked about their experiences in the program. 

OBR provided student transcript data that were used for the analysis of academic out- 
comes such as persistence, credits attempted, and credit accumulation for all public Ohio 
colleges. OBR also provided data on degrees earned by students for all public Ohio colleges. 
In addition, National Student Clearinghouse (NSC) data were obtained to provide additional 
infonnation on institutions not covered by the OBR . 10 Finally, employment data were ob- 
tained from the Ohio Department of Job and Family Services (ODJFS) and are used to 
measure labor market outcomes. 

Students in the Sample 

Using internal financial aid databases, the colleges identified eligible first-time and con- 
tinuing students, and invited them by e-mail or letter to on-campus infonnation sessions. At 


8 The principles of this design are discussed in Bloom (1984). 

"’Sinclair students in the fall 2008 and winter 2009 cohorts had the survey administered during the third 
program quarter, the spring 2009 and fall 2009 quarters, respectively. 

10 The NSC, a nonprofit organization, collects and distributes enrollment, degree, and certificate data from 
more than 3,600 colleges that enroll more than 98 percent of the nation’s college students. See “About the 
Clearinghouse” at www.studentclearinghouse.org. 
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these information sessions, students learned about the program and the potential changes in their 
financial aid packages if they were to join the study and be assigned to the program group. After 
the session, interested students gave their written informed consent to participate, filled out a 
BIF, and supplied their contact information. Finally, the 2,285 students who completed the 
process were randomly assigned. 11 

Table 1 shows that the majority of the sample members, who were nearly all parents, 
were single mothers and that they were generally older students (the average age was about 30), 
with more than one child. Most students had earned a high school diploma or a General Educa- 
tional Development (GED) certificate more than five years before entering the study (69 
percent). About 55 percent of the students were white, 30 percent were black, and 9 percent 
were Hispanic. Nearly 70 percent of the students had someone in their household receiving 
some government benefits. About half of the students were employed at the time of random 
assignment, with 60 percent of those employed reporting that they worked 20 hours per week or 
more (not shown in table). On average, students had an adjusted gross income of just over 
$10,000 per year. 

The fourth column in Table 1 reports differences between the program and control 
groups on average baseline characteristics and shows that the groups look essentially the same 
on the measured attributes: Very few differences between the groups are statistically significant, 
and a likelihood ratio test used to test whether baseline characteristics jointly predicted research 
group status yields a p-value of 0.98. 12 


Program Implementation 

In general, the program was implemented as designed. Monitoring of perfonnance and award 
disbursement for the scholarships found that each of the colleges demonstrated a strong capacity 
to implement the program properly. Additionally, the colleges successfully communicated with 
students about the program in six reminders over the duration of the program. 


Impact Estimates 

The performance-based scholarship program in Ohio was designed to increase students’ 
financial aid, alter students’ attitudes about education, and provide positive incentives to change 
their academic behavior, with the ultimate goal of improving their academic and employment 


1 1 Students had a 60 percent probability of assignment to the program. 
12 For additional details, see Cha and Patel (2010). 
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Table 1: Selected Characteristics of Sample Members at Baseline 


Characteristic 

Full 

Sample 

Program 

Group 

Control 

Group Difference 

Standard 

Error 

Female (%) 

86.4 

86.6 

86.2 

0.4 

1.45 

Unmarried (%) 

78.8 

78.5 

79.3 

-0.8 

1.78 

Parent (%) 

99.7 

99.7 

99.8 

-0.1 

0.22 

More than one child (%) 

59.0 

58.7 

59.4 

-0.7 

2.09 

Single mother (%) 

66.6 

66.3 

67.0 

-0.7 

2.01 

Average age 

29.9 

29.9 

29.9 

0.0 

0.34 

Race/ethnicity a (%) 






White 

54.6 

55.5 

53.2 

2.3 

2.12 

Black 

31.4 

30.3 

33.0 

-2.7 

1.95 

Flispanic 

8.6 

8.1 

9.4 

-1.3 

1.18 

Asian or Pacific Islander 

0.6 

0.6 

0.7 

0.0 

0.34 

Other 

4.8 

5.5 

3.7 

1.8 * 

0.91 

Flousehold receiving any government benefits b (%) 

68.5 

67.7 

69.7 

-2.1 

2.01 

Currently employed (%) 

48.8 

47.6 

50.6 

-3.0 

2.14 

Date of high school graduation/GED certificate receipt 

(%) 





During the past 5 years 

25.3 

25.2 

25.3 

-0.1 

1.85 

More than 5 years ago 

68.8 

68.6 

69.1 

-0.6 

1.96 

Missing 1 " 

6.0 

6.2 

5.5 

0.7 

1.00 

Average Adjusted Gross Income ($) 

10,317 

10,530 

9,999 

531 

383 

Sample size 

2,285 

1,359 

926 




SOURCES: MDRC calculations using Baseline Information Form and Free Application for Federal Student 
Aid data. 

NOTES: To analyze whether baseline characteristics jointly predicted research group status, a likelihood 
ratio test was performed. This test yielded a p-value of 0.98. 

Distributions may not add to 100 percent because of rounding. 

Missing values are only included in variable distributions for characteristics with more than 5 percent of 
the sample missing. 

Respondents who said they are Flispanic and chose a race are included only in the Flispanic category. 
Respondents who said they are not Flispanic and chose more than one race are considered multiracial. These 
respondents, combined with those who said they are American Indian /Alaskan Native or another 
race/ethnicity, are included in Other. 

b Benefits include unemployment/dislocated worker benefits, Supplemental Security Income or disability, 
cash assistance or welfare, food stamps, and Section 8 or public housing. 

c Missing includes students who did not graduate high school and students who did not provide a 
graduation date. 





outcomes. The randomized design used for this study provides strong, reliable estimates of the 
program’s impact on each of these categories of outcomes. 

Financial Aid 

Students were paid money from the scholarships they earned in addition to any finan- 
cial aid they had already been awarded. 13 During the first year of the program, the 2008-2009 
academic year, Ohio offered relatively generous need-based financial aid to community college 
students, and the Ohio College Opportunity Grant (OCOG) was the main vehicle for that aid. 
The maximum Pell Grant — the primary federal source of financial aid — was $4,73 1, and the 
maximum OCOG grant was $2,496. The actual amounts of the need-based awards that a 
student received were affected by the student’s cost of attendance (COA), Expected Family 
Contribution (EFC), and enrollment status (full time or part time). A student’s COA includes 
estimates for costs related to, for example, tuition, fees, books, transportation, and living 
expenses. 14 In the 2009-2010 academic year, however, the amount available in Ohio for need- 
based financial aid declined precipitously. The OCOG program was restructured and received a 
smaller allocation after severe budget constraints in 2009, and OCOG eligibility for community 
college students was terminated. The first cohort of the program was unaffected by this change 
during the program’s duration (but would lose their OCOG after the program ended). Later 
cohorts saw their OCOG awards eliminated during the program, and the perfonnance-based 
scholarship represented a larger proportion of their financial aid packages. 

Table 2 shows financial aid outcomes for the program year for all students and demon- 
strates that the program effectively increased financial aid receipt for students in the program 
group. The first row of the first panel of Table 2 shows that a large majority of students received 
financial aid during their first academic year in the study — about 95 percent of students in each 
of the groups. The second row, however, shows that during this same year, 73.8 percent of 
program group students also received a perfomiance-based scholarship, while students in the 
control group were not eligible for the scholarship and consequently did not receive it. The 
remainder of the first panel shows that approximately equal percentages of students in the 
program group and control groups received loans, Pell Grants, and other grants. 

The second panel of Table 2 shows the average aid amounts that students received for 
each type of aid. The first row of the second panel of Table 2 shows that on average, program 
group students received an estimated $7,947 in total financial aid, while students in the control 
group received an estimated $7,445: The estimated impact of the program on the average 


l3 ln some cases, the scholarship allowed students to reduce the loans in their financial aid packages. 
14 The information in this section is adapted from Cha and Patel (2010). 
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Table 2: Impacts on Financial Assistance During the First Academic Year 


Outcome 

Program 

Group 

Control 

Group Difference 

Standard 

Error 

Academic Year 1 





Received any financial assistance (%) 

95.5 

94.8 

0.7 

0.90 

Received performance-based scholarship' 1 

73.8 

-0.1 

72 9 *** 

1.43 

Received loans 

58.3 

57.8 

0.5 

1.96 

Received Pell Grant 

90.9 

91.2 

-0.2 

1.18 

Received any other grants' 3 

73.0 

73.2 

-0.2 

1.81 

Average financial assistance received ($) 

7,947 

7,445 

502 *** 

185 

Performance-based scholarship 11 

765 

-1 

766 *** 

20 

Federal loans 

2,853 

3,187 

-334 *** 

120 

Pell Grant 

3,395 

3,336 

59 

74 

Other grants' 1 

934 

923 

11 

36 

Sample size (total = 2,285 ) 

1,359 

926 




SOURCES: MDRC calculations using financial aid data provided by Lorain County Community College, 
Owens Community College, and Sinclair Community College. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by cohort and campus. 

Rounding may cause slight discrepancies in sums and differences. 

Federal work study awards are excluded from the figures above due to unavailability of data. 
a The estimate of -$1 for students in the control group is due to regression adjustments. Only program 
group students received performance-based scholarship awards during the program period. 

b This category includes all grants and scholarships excluding Pell Grant and performance-based 
scholarship awards. 


amount of financial aid package is $502, statistically significant at the 0.01 level. The second 
and third rows of the same panel show how the program appears to have caused this difference. 
The second row shows that students in the program group are estimated to have received an 
additional $766 from the scholarship, a substantial increase, but still just about 10 percent of the 
average financial assistance students received during the program year. The third row shows 
that students in the program group took out less money in loans, an estimated reduction of $334. 
This loan reduction was in part a result of a reduction in unsubsidized loans, which are less 
advantageous to students. While students are in school, the government pays the interest on 
subsidized loans, but not unsubsidized loans. Students in the program group reduced their 
unsubsidized loans by an estimated average of $147 (not shown). The scholarship was not 
specifically geared to reduce loans, but the reduction in educational debt may be an added 
benefit for students in the long run. 
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Hypothesized Mediators of Academic Success 

The opportunity for additional aid was hypothesized to motivate students academically, 
promote behaviors believed to help students graduate, such as enrolling full time and studying 
more in order to meet the academic benchmarks, and instill confidence in students’ ability to 
succeed. The award was also intended to reduce students’ financial stress and increase the 
amount of time they could dedicate to schoolwork and studying, by providing financial aid that 
could reduce the need to work. Table 3 shows results of the survey that was conducted to 
measure student attitudes and changes in behavior related to these hypotheses. In all, 1,096 
students in the program group and 745 students in the control group completed the survey. 15 

In general, the survey results suggest that the program did not substantially change stu- 
dents’ attitudes and behaviors, but may have done so to a modest degree. The first row of Table 
3 shows that students in the program group did not report spending more time studying for their 
most recent exam or assignment than their control group counterparts: Students in the program 
group reported an average of 8.2 hours, compared with 8.4 hours in the control group, a differ- 
ence that is not statistically significant. Table 3, however, also shows that the program may have 
helped students feel more prepared for exams or assignments and increased their confidence: 

26.2 percent of students in the program group felt they prepared extremely well compared with 

22.3 percent of students in the control group, a 3.9 percentage point difference — significant at 
the 10 percent level. 

The survey also asked students about their motivation to complete their course load, us- 
ing several questions that were aggregated into a relative autonomy index (RAI). The RAI is an 
overall measure of motivation that ranges from -18 to 18, with higher values representing 
greater autonomous or personally driven motivation. Research suggests that greater autonomous 
motivation leads to improved outcomes such as achievement (Ratelle, Guay, Vallerand, Larose, 
and Senecal, 2007). The fourth row of Table 3 shows that students in both groups had similar 
levels of RAI (an average of 3.9 in each group), suggesting the program did not increase 
motivation during the program year. It also does not appear to have negatively affected motiva- 
tion, a potential concern related to perfomiance-based payments (Deci, Koestner, and Ryan, 
2001). 16 


15 Students in the program group did not differ systematically from students in the control group, as meas- 
ured by observable baseline characteristics: An omnibus F-test of respondents by treatment group yielded a 
p-value of 0.9720. An omnibus F-test was also performed to compare survey nonrespondents with survey 
respondents. The test yielded a p-value of 0.001, suggesting there are some statistically significant differences 
in baseline characteristics between respondents and nonrespondents, but the respondents constituted a large 
proportion of the study sample, so nonrespondents were a minority: Across all the colleges and cohorts, the 
response rate was between 79 percent and 82 percent. 

16 The survey cannot assess whether motivation changed in later semesters or quarters. 
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Table 3: Survey Responses: Educational Experiences 


Characteristic 

Number of 
Observations 

Program 

Group 

Control 

Group Difference 

Standard 

Error 

For the most recent exam, final project, or paper: 

Number of hours studied 

1,838 

8.2 

8.4 

-0.3 

0.48 

Did not seek help (%) 

1,841 

52.3 

49! 

3! 

2.37 

Prepared adequately (%) 

Did not prepare adequately 

1,841 

1.8 

3! 

-1.3 * 

0.72 

Prepared a little 

1,841 

2.8 

2.9 

-0.2 

0.79 

Prepared somewhat 

1,841 

23.0 

24.2 

-1.3 

2.01 

Prepared quite a bit 

1,841 

46.3 

47.5 

-1.2 

2.37 

Prepared extremely well 

1,841 

26.2 

22.3 

3.9 * 

2.05 

Motivation to complete coursework (RAI) d (average) 

1,841 

3.9 

3.9 

-0! 

0.17 

Ever asked instructor to reconsider a grade (%) 

1,841 

29.7 

26! 

3.5 * 

2.14 

Sample size 

1,841 

1,096 

745 




SOURCE: MDRC calculations using the performance-based scholarship survey. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by cohort and campus. 

Distributions may not add to 1 00 percent because of rounding. 

a Motivation to complete coursework is defined using the Relative Autonomy Index (RA1), which has a range of 
- 18 to 18. A higher value represents greater autonomous motivation. The RAI is calculated as a weighted average: 
RAI = External*(-2) + lntrojected*(-l) + Identified*!!) + Integrated*!!). 


Finally, the last row of Table 3 suggests that the scholarship program may have caused 
students to be attentive to their grades, since earning the scholarships depended on these grades. 
Table 3 shows that 29.7 percent of students in the program group requested that instructors 
reconsider a grade, while only 26. 1 percent of students in the control group did so, a difference 
of 3.5 percentage points and significant at the 10 percent level. It is possible that more students 
in the program group were simply making these requests as an easy way to improve their 
grades. The survey data, however, show that less than 2 percent of program group students said 
that they took easier classes to meet the GPA requirements. 

Academic Outcomes 

A key goal of the program was to improve students’ academic achievement. Table 4 
provides strong evidence that the program did have an effect on important academic outcomes. 
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Table 4: Impacts on Academic Outcomes During the First Four Academic Years 


Outcome 

Program 

Group 

Control 

Group Difference 

Standard 

Error 

Full-time completion* (%) 

Term 1 

33.2 

26.3 

7 o *** 

1.94 

Term 2 

28.8 

18.3 

10.5 *** 

1.80 

Ever registered (%) 

Year 1 

96.9 

95.9 

1.1 

0.78 

Year 2 

68.8 

66.6 

2.3 

1.97 

Year 3 

48.8 

50.2 

-1.5 

2.12 

Year 4 

34.7 

33.4 

1.3 

2.00 

Credits attempted 1 * 

Year 1 

21.5 

20.6 

0 9 ** 

0.40 

Year 2 

13.4 

12.5 

0.9 * 

0.50 

Year 3 

00 
0 6 

8.7 

0.1 

0.47 

Year 4 

5.6 

5.5 

0.1 

0.40 

Cumulative credits attempted after four years 

49.3 

47.3 

2.0 

1.35 

Credits earned 1 * 

Year 1 

15.9 

14.2 

j 7 *** 

0.43 

Year 2 

9.6 

8.9 

0.7 

0.44 

Year 3 

6.4 

6.4 

0.1 

0.41 

Year 4 

3.9 

4.0 

0.0 

0.33 

Cumulative credits earned after four years 

35.8 

33.4 

2.5 * 

1.25 

Sample size (total = 2,285) 

1,359 

926 




SOURCE: MDRC calculations from the Ohio Board of Regents transcript data from all public Ohio 
institutions. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by cohort and campus. 

Rounding may cause slight discrepancies in sums and differences. 

a Full-time completion is defined as earning 12 or more credits at the institution of enrollment. Term 
refers to fall or spring at semester institutions and fall, winter, or spring at quarter institutions. Third 
quarter outcomes are not shown for Sinclair students. 

b Quarter institutions have three quarters per academic year, while semester institutions have two 
semesters. To adjust for inflation due to the additional term at quarter institutions, credits attempted and 
earned at quarter institutions are multiplied by two-thirds. 
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The first panel of Table 4 shows that the program had a relatively large, statistically significant 
effect on the percentage of students who earned a full-time course load of credits (12 credits or 
more) in both the first and second semesters or quarters of the program, indicators of the 
likelihood of completion found in previous research (Adelman, 2006). 17 In the first semester or 
quarter of the program, the estimated impact on the percentage of students who earned full-time 
credits is 7.0 percentage points, and in the second semester or quarter, the estimated impact 
grew to 10.5 percentage points. These estimates suggest that students may have responded 
positively to the differential scholarship structure and the added financial incentive to earn full- 
time credits, and that scholarship payments may have helped students afford more classes in the 
second semester or quarter. 

The second panel of Table 4 shows registration rates for each group for each year of the 
study. Consistent with previous research on performance-based scholarships, the program does 
not appear to have had an impact on registration in the first year or on persistence in subsequent 
years. In the first year, more than 95 percent of students in both groups registered for classes, 
with no statistically significant differences between the groups. During all follow-up years, both 
program and control group students enrolled at essentially equal rates. By the fourth year, only 
34.7 percent of program group students and 33.4 percent of control group students were still 
registering for classes. 

In contrast, the third panel of Table 4 shows that despite enrolling at nearly equal rates, 
students in the program group took more classes than students in the control group. In the first 
year, students in the program group attempted an average of 21.5 credits, compared with an 
average of 20.6 credits in the control group — a difference of 0.9 credits that is statistically 
significant at the 5 percent level. In the second year, program group students attempted 0.9 
credits more than their control group counterparts, a difference significant at the 10 percent 
level. In the third and fourth years, however, students in the program group only attempted an 
average of 0. 1 credits more than the control group, a difference that is not statistically signifi- 
cant. These results suggest that the program was successful in motivating students to attempt 
more credits during program semesters and that students modified their behavior in the short 
term. Results from the survey support this finding. In the program group, 72.9 percent of 
respondents indicated that the scholarship encouraged them to take more or continue taking 
classes. 


The fourth panel of Table 4 shows that the positive estimate for credits attempted is as- 
sociated with a positive estimate for credits earned. On average, students in the program group 
earned more credits compared with students in the control group. In the first year, students in 


17 Adelman (2006) argues that earning less than 20 credits in the first year is a “serious drag” on 
completion. 
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the program group earned an average of 15.9 credits, compared with an average of 14.2 credits 
in the control group — an estimated impact of 1.7 credits, or a nearly 12 percent increase, 
statistically significant at the 1 percent level. Furthermore, the estimated impact on credits 
earned is larger than that on credits attempted in the first year, suggesting that students in the 
program group may have also been doing better in classes and not just taking more classes. In 
the second year of the study, after the scholarship ended, program group students earned an 
average of 0.7 credits more than control group students, but the difference is not statistically 
significant. In the third and fourth years, both groups earned the same number of credits on 
average. Differences in credits earned each year after the first year are not statistically signifi- 
cant. The last row of Table 4 shows that after four years of follow-up the estimated impact of 
the program increased to 2.5 credits — a cumulative difference that is statistically significant at 
the 10 percent level. The estimates during the first year, the program year when students could 
earn the scholarship, account for two-thirds of the cumulative impact estimate. 

These results suggest that despite the modest effects measured by the survey, the pro- 
gram did change student behavior: More program than control group students met full-time 
credit benclnnarks and program students earned more credits overall. 

Degrees 

The program also appears to have produced short-term impacts on degree completion, 
and accelerated the time it took students to earn degrees. Figure 1 shows the percentages of 
students in the program and control groups who had earned degrees at the end of each follow-up 
year. Less than 10 percent of students in both groups earned a certificate or degree in the first 
year. In the second year of the study, however, 20.5 percent of students in the program group 
had done so, compared with 17.2 percent of students in the control group — an estimated 
impact of 3.3 percentage points, significant at the 5 percent level. By the end of the third year, 
28.3 percent of students in the program group had earned a certificate or degree compared with 
24.5 percent in the control group — a difference of 3.8 percentage points, also significant at the 
5 percent level. But by the end of the fourth year, evidence of an impact disappears; the gradua- 
tion rate in the control group increased more than the graduation rate in the program group. In 
the program group, 34.6 percent of students had earned a degree compared with 32.9 percent of 
students in the control group — a difference of 1.6 percentage points that is not statistically 
significant. 18 

These results suggest that the program may have shortened the time it took students to 
earn a credential. Although the results do not provide evidence of a long-term impact on degree 


ls Rounding may cause slight discrepancies in sums and differences. 
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Figure 1: Certificate or Degree Attainment After Four Years 



Y ear 1 Y ear 2 Y ear 3 Y ear 4 


■ Program group □ Control group 

SOURCES: MDRC calculations using Ohio Board of Regents and National Student Clearinghouse degree data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 
5 percent; * = 10 percent. Standard errors are shown in parentheses. 

Estimates are adjusted by cohort and campus. 

Rounding may cause slight discrepancies in sums and differences. 



completion, the estimated impacts on credit accumulation after four years show that the pro- 
gram’s impacts on students’ academic outcomes remained evident even three years after the 
program ended. 

Employment Outcomes 

As discussed earlier, the program also aimed to reduce students’ need to work in the 
short term, while in school. In the long term, the program aimed to improve earnings and 
employment outcomes by helping students earn degrees and enter the workforce earlier and 
more prepared. Table 5 shows measures of student employment during and after the program. 

Table 5 shows that students in the program and control groups were employed at simi- 
lar rates and earned similar amounts throughout the four years of follow-up. During the first 
year after random assignment, for example, 62.1 percent of students in the program group were 
employed, compared with 64.5 percent of students in the control group. 19 The 2.5 percentage 
point difference is not statistically significant. This result also holds when measured by the 
percentage of students who were ever employed during the follow-up period and the total 
number of fiscal quarters during which students were employed in the follow-up period. 20 Table 
5 also provides little evidence that the program had a long-term impact on earnings: Students in 
the program and control groups generally earn similar amounts during all the follow-up years. 21 

These results are unsurprising given the lack of detectable impacts on degree receipt af- 
ter four years. They also align with recent research that suggests detectable impacts on em- 
ployment outcomes may be too much to expect for programs that last for relatively short 
durations and do not produce exceptionally large academic impacts (Weiss et ah, 2014). 


Discussion 

Overall, this study helps fill an important gap in the literature on financial aid programs and 
their long-term effectiveness. It used a randomized controlled trial with over 2,000 students to 
evaluate an important financial aid program, and it includes four years of post-random 


19 To account for variation in the timing of random assignment, the first year of employment data omits the 
fiscal quarter of random assignment. 

20 Self-reported employment responses in the survey also corroborate this result. 

21 Table 5 shows that the average earnings for students in the first-year program group with jobs covered 
by unemployment insurance were $6,073, compared with $6,496 in the control group, a reduction of $423 that 
is not statistically significant. When this same estimate is calculated including regression adjustments for 
earnings and employment from the three years prior to random assignment as covariates (not shown), the 
estimate increases to a reduction of $540, significant at the 10 percent level. This result may provide some 
support for the hypothesis that the program helped students reduce employment. 
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Table 5: Impacts on Employment and Earnings 


Outcome 

Program 

Group 

Control 

Group Difference 

Standard 

Error 

Employed (%) 

Year 1 

62.1 

64.5 

-2.5 

2.06 

Year 2 

65.6 

64.9 

0.7 

2.03 

Year 3 

69.0 

65.9 

3.1 

1.99 

Year 4 

70.5 

71.0 

-0.4 

1.94 

Ever employed during fiscal quarters 2-20 (%) 

87.0 

88.3 

-1.3 

1.41 

Quarters employed, fiscal quarters 2-20 (%) 

56.2 

55.9 

0.3 

1.55 

Earnings from Unemployment Insurance-covered 
jobs ($) 

Year 1 

6,073 

6,496 

-423 

354 

Year 2 

7,776 

7,751 

26 

431 

Year 3 

9,851 

9,425 

426 

513 

Year 4 

11,710 

11,645 

65 

578 

Total earnings, fiscal quarters 2-20 ($) 

45,474 

45,557 

-82 

1,995 

Sample size ( total = 2,285 ) 

1,359 

926 




SOURCE: MDRC calculations using administrative records from the Ohio Department of Job and 
Family Services. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by cohort and campus. 

Rounding may cause slight discrepancies in sums and differences. 

Employment and earnings estimates do not include the fiscal quarter during which students were 
randomly assigned. 


assignment longitudinal data on degree receipt, employment, and earnings. It is important to 
emphasize that the performance-based scholarship program examined here lasted for only one 
year and provided a relatively modest financial award. The scholarship itself provided about 
10 percent of the average total financial aid received by students in the program group during 
the program year, and a much smaller average percentage over the full duration of students’ 
college careers. Still, the evidence suggests that the program helped students graduate faster 
and earn more credits in the long term. 

The low-income parents in this study also faced the additional challenges of caring for 
children while pursuing their academic studies and helping to support a family. Their odds of 
graduating were low to begin with: Only a third of the students in the control group earned a 
credential within four years, even though some of the students had started college prior to the 
start of the study. This study also reports important findings that build on the accumulating 
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knowledge of the short-term impacts of perfonnance-based scholarships. These findings 
suggest that the scholarships helped more students advance toward their degrees by earning 
more credits during the program, and the results can inform both future research and the 
evolution of scholarship programs. The scholarship structure itself was designed to be flexible 
for the targeted students. Students could earn an award for part-time academic benchmarks, but 
the program also provided an incentive for full-time credit accumulation. In fact, this study finds 
a 10 percentage point impact on full-time credit accumulation after two semesters or quarters. 
Students in the program group also reduced their reliance on loans, including unsubsidized 
loans for which interest accrues even while students remain in school. At the same time, they 
received more overall financial aid. 

Low graduation rates remain a pressing problem in community colleges, especially 
for low-income students, and rigorous evidence about the long-term impacts of most commu- 
nity college programs is still relatively rare. This study provides an important example of how 
a state used existing funds to both help low-income students improve their academic out- 
comes and rigorously evaluate the program’s effectiveness. The program lasted for only one 
year, and the estimated impacts on credit accumulation occurred primarily during the program 
year, but the program’s impacts on academic outcomes were still evident three years after the 
program ended. States and other organizations can build on this model to better understand 
what works for community college students. Future studies can make important contributions 
by incorporating rigorous research designs, studying performance-based scholarship pro- 
grams with longer durations, and examining the impacts on students’ long-term financial 
outcomes and graduation rates. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



